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1.  INTRODUCTION  I 

1 

i 

Operational  military  decision  makers  in  the  western 
North  Pacific  routinely  use  information  on  average  tropical 
cyclone  forecast  errors  in  order  to  determine  actions  required 
to  evacuate,  evade  or  protect  against  storms.  For  example, 

U.S.  military  staff  meteorologists  incorporate  average  fore- 
cast error  statistics  in  their  decision-making  recommendations 
of  typhoon  conditions^  for  Department  of  Defense  installations. 

These  recommendations  therefore  impact  the  planning  for  the 
evacuation  of  aircraft  from  a threatened  base  or  the  level  of 
base  shutdown  preparedness.  Average  error  statistics  are 
also  used  in  regards  to  decisions  concerning  ships  whether  in 
port  or  evading  at  sea.  Because  of  the  number  of  tropical 
cyclones  traversing  the  western  North  Pacific  each  year 
(approximately  35),  and  the  large  number  of  ports  and  bases 
and  installations  used  directly  and  indirectly  by  the  U.S. 
military  which  may  be  in  the  paths  of  these  tropical  cyclones, 
such  decisions,  whether  right  or  wrong,  are  made  frequently. 

Average  forecast  error  information  currently  available 
does  not  provide  the  most  accurate  decision-making  inputs 
now  possible.  If  more  accurate  information  was  available  it 
could  be  of  significant  value  to  Department  of  Defense 
decision  makers,  and  could  provide  valuable  guidance  on 
optimum  evasion  and  evacuation  tactics. 

It  should  be  emphasized  ti»at  decision  lead  times  for 
ships  are,  of  course,  much  longarethan  for  aircraft  evacuation 
or  base  preparedness.  Ships  have  to  get  underway  well  before 
winds  and  seas  start  to  affect  significantly  their  ability  to 


^Conditions  of  readiness  relating  to  expected  destructive 
winds . 


maneuver  clear  of  a harbor  and  thereafter  their  speed-of- 
advance.  Since  decision  errors  are  a function  of  forecast 
errors  which  increase  as  a function  of  time  between  forecast 
and  verification,  a larger  number  of  incorrect  decisions 
could  be  expected  for  ships  than  for  aircraft  evacuations  or 
base  closures. 

Thus  the  need  exists  for  an  analysis  of  the  variations 
in  the  western  North  Pacific  forecast  performance  for  differ- 
ent areas,  times  of  the  year  as  well  as  for  those  tropical 
cyclones  having  recognizably  different  characteristics.  This 
study  attempts  to  do  such  an  analysis  with  the  following 
goal s ; 

(a)  To  develop  a tropical  cyclone  forecast  error  data 
set  to  provide  a basis  for  statistical  analysis  of  past 
errors  and  provide  a benchmark  for  future  error  improvements. 

(b)  To  identify  situations  where  the  forecasts  are  very 
good  or  very  bad  in  order  to  allow  maximum  concentration  of 
resources  for  quick  reduction  of  the  largest  errors. 

(c)  To  provide  probability  algorithms  for  an  estimate 
of  the  forecast  errors  of  warnings  in  order  to  assist  western 
Pacific  commanders  in  operational  decisions  regarding  the 
protection  and  evacuation  of  military  resources. 

(d)  To  stratify  errors  for  24-,  48-,  and  72-hour  fore- 
casts based  on  various  parameters  such  as  location,  time  of 
year,  and  various  tropical  cyclone  characteristics. 

(e)  To  determine  if  the  year  to  year  variations  in 
forecast  accuracies  are  real  or  random  deviations  about  a long 
term  mean  and,  if  real,  determine  the  reasons  for  the 
variations . 


2.  DATA  AND  METHOD  OF  ANALYSIS 

The  Joint  Typhoon  Warning  Center  (JTWC),  Guam  provided 
10  years  (1966-1975)  of  forecast  and  "best  track"^  tropical 
cyclone  information  for  the  basic  statistical  analysis.  The 
data  set  included  the  24-,  48-  and  72-hour  official  forecasts 
issued  by  the  JTWC.  6150  six-hourly  best  track  positions 
for  317  tropical  cyclones  (including  depressions,  tropical 
storms  and  typhoons)  were  examined.  There  were  some  storms, 
however,  that  were  so  short-lived  as  to  provide  no  verifying 
forecasts.  These  were  not  represented  in  the  data  set  that 
was  statistically  analyzed.  For  instance,  to  verify  an  error 
distance  for  the  24-hour  forecast,  a best  track  must  be  avail- 
able 24  hours  later.  If,  in  that  24-hour  period,  the  storm 
dissipated  and  was  no  longer  identified  by  a best  track 
position,  the  forecast  could  not  be  verified.  This  also 
accounts  for  the  fact  that  fewer  cases  were  verified  for 
72-hour  forecasts  than  for  48  hours,  and  fewer  48-hour  than 
24-hour  forecasts.  The  verifying  cases  totaled  as  follows; 


24  hours 
48  hours 
72  hours 


4809  forecasts 
3038  forecasts 
1372  forecasts 


The  foil 

owing  parameters  were  also  analyzed  for  each 

forecast 

i ni ti ati 

on  time: 

1 . 

Maximum  wind 

2. 

Lati tude 

3. 

Longitude 

4. 

West-east  component  (positive  to  the  east) 
tropical  cyclone  movement 

of 

5. 

South-north  component  (positive  to  the  north)  of 
tropical  cyclone  movement 

poBt-anatyaia  aet  of  poaitiona  and  maximum  winda  baaed 
on  all  available  information. 
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6.  Position  number  on  storm  track  (related  to  warning 
number  and  1 ength  of  existence  of  storm) 

7.  Number  of  storms  in  progress  at  forecast  time 

8.  Month 

9.  Error  distance  (forecast  position  to  best  track 
position) 

10.  Direction  from  forecast  position  to  best  track 
posi ti on 

The  1976  data  was  processed  in  a similar  manner,  but 
retained  separately  for  independent  testing.  In  this  data 
set  there  were  625  best  track  positions  at  six-hourly  inter- 
vals for  25  tropical  cyclones  and  the  verifying  cases  totaled: 

24  hours  - 524  forecasts 
48  hours  - 424  forecasts 
72  hours  - 332  forecasts 

The  1966-1975  basic  data  set  was  then  examined  by  stepwise 
multiple  linear  regression  and  discriminant  analysis  techniques 
to  determine  basic  statistical  relationships  between  parameters 
and  find  parameters  related  to  forecast  errors.  The  results 
were  then  tested  on  the  1976  data  set.  The  results  of  the 
statistical  analysis  were  also  used  to  examine  the  year-to-year 
variation  in  forecast  accuracy  as  well  as  the  trend  during  the 
11-year  period. 


twmm 
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3.  DISCUSSION  OF  RESULTS 

3.1  Linear  Regression  and  Correlation  Analysis 

The  initial  statistical  analysis  of  the  variables  employed 
the  UCLA  Biomedical  computer  program  BMD02R  stepwise  multiple 
linear  regression  (Dixon,  1970).  Tables  1 and  2 summarize  the 
means,  standard  deviations,  variances  explained,  and  the 
correlation  matrix  of  the  first  nine  variables  discussed  above 
for  the  24-,  48-  and  72-hour  forecasts.  The  maximum  coeffi- 
cient of  any  available  predictor  on  the  magnitude  of  the 
forecast  error  (at  either  24,  48  or  72  hours)  was  0.185. 
However,  many  of  the  correlations  were  significant  on  the  ^% 
level.  The  maximum  total  explained  variance  of  the  error 
distance  was  less  than  11%.  The  variables  contributing  most 
to  the  explained  variance  were  the  maximum  wind,  latitude, 
longitude,  west-east  movement  and  south-north  movement.  The 
concept  of  predicting  the  error  was  then  oriented  toward  a 
discriminant  analysis  where  forecasts  could  be  identified,  and 
hopefully  forecast,  as  either  "good"  or  "bad,"  This  will  be 
discussed  later. 

3.2  Trends 

In  order  to  examine  the  forecast  errors  in  more  detail, 
the  errors  were  stratified  and  mean  errors  were  computed  for 
each  stratification  along  the  range  of  each  variable.  Signifi- 

7 

cant  trends  were  evident.  In  the  analysis,  stratifications 
were  selected  to  keep  the  number  of  cases  in  each  group 
relatively  high.  Frequencies  are  typically  a few  hundred  and 
are  not  indicated  except  where  they  drop  below  100.  As  an  aid 
in  interpreting  the  forthcoming  figures,  relative  frequencies. 


3 

Trends  to  be  discussed  were  subjected  to  "t"  tests  and 
most  were  shown  significant  at  least  at  the  5%  level  with 
some  at  the  1%  level. 
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Table  1.  Basic  statistical  information  of  the  stepwise 
multiple  linear  regression  analysis  (1966-1975  data) . 


MEANS 


VARIABLE 


MAXIMUM  WIND 

ms~^ 

(kt) 


LATITUDE  (°N) 


LONGITUDE  (°E) 


WEST-EAST  MOVT 

ms"^ 

(kt) 


SOUTH-NORTH  MOVT 

ms”^ 

(kt) 


POSITION  NUMBER 


NUMBER  OF  STORMS 


MONTH 


24-HR  FORECAST 
ERROR: 

km 

(n  mi ) 


48-HR  FORECAST 
ERROR: 

km 

(n  mi ) 


72-HR  FORECAST 
ERROR: 

km 

n mi 


TOTAL  VARIANCE  EXPLAINED  {%) 


STANDARD 

DEVIATION 


VARIANCE  EXPLAINED  Ul 
24-HR  48-HR  72-HR 


NE:  Was  rejected  in  linear  regression 
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Table  2.  Correlation  matrix  for  the  24-,  48-  and  72-hour  forecast  error  data  (1966-1975) 


4->  O <0 
<0 

01 

t-  f—  O 

s-  o 
o o s- 

O 4->  CO 
* 


all  based  on  4809  cases,  are  shown  for  the  24-hour  forecasts. 
Since  they  are  in  percentages  of  the  total,  the  distributions 
roughly  apply  also  for  48-  and  72-hour  forecasts. 

3.2.1  Geographic  Position 

In  each  of  the  24-,  48-  and  72-hour  forecast  situa- 
tions, the  mean  forecast  errors  were  minimal  at  lower 
latitudes  (Fig.  1(a)),  gradually  increasing  with  latitude. 

This  indicates,  as  expected,  that  storms  are  more  accurately 
forecast  before  they  recurve  and  move  into  higher  latitudes. 
Mean  forecast  errors  decrease  with  decreasing  east  longitude 
(Fig.  1(b)).  Generally,  a forecast  for  a storm  in  a more 
westerly  position  is  one  based  on  a longer  than  average 
history,  and  is  in  an  area  of  better  synoptic  data  and  land 
radar  coverage,  given  the  proximity  to  the  Philippines,  Taiwan, 
and  China,  and  other  continental  areas  west  of  130E. 

The  geographic  variations  of  mean  24-  and  48-hour  fore- 
cast errors  can  be  seen  in  Fig.  2.  This  figure  dramatically 
shows,  for  example,  the  difference  between  tropical  cyclone 
forecast  errors  for  tropical  cyclones  affecting  the  Philippines 
versus  those  affecting  the  Japan/Korea  area. 

3.2.2  Maximum  Wind 

Maximum  wind  is  another  important  parameter.  As 
shown  in  Fig.  3,  the  mean  errors  decrease  with  increasing 
maximum  wind  speeds,  indicating  that  better  developed  tropical 
cyclones  are  more  accurately  forecast.  It  should  be  noted 
that  better  developed  tropical  cyclones  are  generally  ones 
with  longer  histories  and  more  accurate  center  locations. 

These  influences  will  be  discussed  later.  This  general  trend 
between  forecast  error  and  maximum  wind  is  visible  for  all 
three  forecast  times. 


- 8 - 


Figure  2.  Geographic  distribution  of  mean  24-hour  (a)  and 
48-hour  (b)  forecast  errors  for  western  North  Pacific 
tropical  cyclones.  Errors  are  based  on  1966-1975  data 
and  relate  to  mean  forecasts  from  initial  positions. 
Values  are  given  in  km  ( ) and  n mi. 
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MEAN  FORECAST  ERROR 


3.2.3  Direction  of  Movement 

Forecasts  are  generally  better  for  tropical  cyclones 
moving  west  (Fig.  4(a))  and  become  progressively  more  diffi- 
cult as  westward  movement  diminishes  and  becomes  eastward  as 
associated  with  recurvature.  As  for  the  south-north  component 
(Fig.  4(b)),  the  best  forecasts  are  centered  at  or  near  zero, 
again  implying  better  forecasts  when  the  storm  is  moving  west 
with  little  or  no  deflection  north  or  south.  Errors  increased 
markedly  for  storms  moving  south,  as  well  as  for  tropical 
cyclones  moving  north,  as  would  be  associated  with  recurvature. 

3.2.4  Multiple  Storms 

Figure  5(a)  shows  a relationship  between  number  of 
tropical  cyclones  occurring  simultaneously  and  an  increase  in 
forecast  error.  This  is  reinforced  by  Fig.  5(b)  which 
examines  distance  to  the  nearest  storm  as  a function  of  fore- 
cast error.  With  tropical  cyclones  in  close  proximity  (less 
than  1112  km  [600  n mi])  forecast  errors  increase  dramatically. 
The  time  of  year  also  relates  to  number  of  tropical  cyclones 
occurring  simultaneously  and  the  monthly  variation  of  forecast 
errors  can  be  seen  in  Fig.  5(c).  The  peak  months  of  tropical 
cyclone  activity  produces  generally  larger  errors  at  24  and 
48  hours  with  large  variations  occurring  in  the  72-hour  errors 
due  in  part  to  the  smaller  sample  sizes  for  this  forecast  time 
period. 

3.2.5  Position  Number  and  Initial  Position  Accuracy 

The  length  of  storm  history,  as  measured  by  the 
position  number,  showed  a trend  congruous  with  that  of  maximum 
wind  for  the  24-  and  48-hour  forecasts.  As  shown  in  Fig.  6(a), 
as  the  storm's  history  (and  therefore  development)  increased, 
the  forecast  errors  decreased.  This  error  trend  is  strongest 
in  the  24-hour  situation,  decays  for  48  hours  and  is  not 
apparent  at  all  for  72-hour  forecasts  as  might  be  expected 
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MEAN  FORECAST  ERROR 


b.  South-North  MOVEMENT 


NUMBER  OF  CASES  PER  STRATIFICATION  SHOWN  WHEN  LESS  THAN  100 


MEAN  FORECAST  ERROR 


Figure  5.  Mean  error  stratifications  by  number  of  tropical 
cyclones  occurring  simultaneously  (a) ; distance  to  the 
nearest  storm  (b)  ; and  month  (c)  . 
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MEAN  FORECAST  ERROR 


MEAN  ERROR  STRATIFICATIONS 


a.  POINT  ON  TRACK 


b.  INITIAL  POSITION  ERROR 


NUMBBR  OF  CASFS  PER  STRATIFICATION  SHOWN  WHEN  LESS  THAN  100 


72 

HOUR 


48 

HOUR 


- 24 
HOUR 


24  HOUR  FREQUENCY  DISTRIBUTION  ( % ) 


-20  40- 

0 19  37  56  74 


9 13  17  21  >23 

POS  NO 


>50  nml 
>93  km 


POS  ERR 


Figure  6.  Mean  error  stratifications  by  point  on  track 
(position  nuinber  which  is  related  to  life  cycle  or 
length  of  existence  of  tropical  cyclone)  (a)  and 
initial  position  error  (b) . 

- 15  - 
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with  tropical  cyclone  recurvature  occurring  late  in  the 
storm  history. 

The  last  variable,  initial  position  error  was 
evaluated  using  initial  "warning  position"^  data  available 
from  1971-1975.  The  consistent  and  prominent  trend  shown  in 
Fig.  6(b)  documents  that  the  mean  forecast  error  increases  as 
the  initial  position  errors  increase.  This  supports  the  basic 
forecasting  premise  that  accurate  observations  are  necessary 
for  accurate  forecasts.  This  finding  is  in  general  agreement 
with  the  findings  in  the  Atlantic  of  Neumann  and  Hope  (1972), 
Neumann  (1975),  and  Sanders  and  Gordon  (1976),  who  found  that 
tor  Atlantic  hurricanes,  the  initial  position  error  was 
important  in  objective  forecasts. 

The  slope  of  the  relationship  of  initial  position 
error  to  forecast  error  is  interesting.  A 23.2  km  (12.5  n mi) 
initial  position  error  relates  to  a 37.1  km  (20  n mi)  fore- 
cast error,  approximating  the  24-hour  slope.  A strong 
relationship  also  exists  at  48  hours  and  even  at  72  hours. 

This  is  an  important  consideration  since  different  reconnais- 
sance platforms  have  different  initial  position  errors.  For 
example,  the  mean  initial  position  error  of  the  Defense 
Meteorological  Satellite  Program  (DMSP)  in  1976  for  the  western 
North  Pacific  was  56.5  km  (30.5  n mi),  which  was  the  basis  for 
some  30.4%  of  the  tropical  cyclone  warnings  issued  by  the  JTWC . 
Whereas,  the  mean  initial  position  error  of  aircraft  recon- 
naissance for  1976  in  the  western  North  Pacific  was  32.8  km 
(17.7  n mi),  which  was  the  basis  for  some  44.3%  of  the  tropical 
cyclone  warnings.  This  will  be  discussed  in  more  detail  later. 


A 

The  "warning  position"  is  the  pos^t^on  the  forecaster 
feels  the  tropical  cyclone  is  located  at  time  of  issuance  of 
the  forecast . The  difference  between  the  Warning  position 
and  the  post  analysis  best  track  position  is  the  initial 
position  error. 
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3.3  Autocorrelations 

To  this  point,  the  forecasts  (and  hence  forecast  errors) 
have  been  tacitly  assumed  to  be  independent  of  each  other. 

In  reality,  successive  six-hourly  forecasts  for  a particular 
storm  are  strongly  correlated.  This  can  be  seen  in  Table  3 
which  gives  the  estimated  autocorrelation  coefficients  between 
errors  from  successive  forecasts  with  lag  times  out  to  36 
hours.  In  general,  it  can  be  seen  that  autocorrelations  at 
any  given  time  lag  increase  as  the  forecast  interval  increases. 


Table  3.  Autocorrelations  between  forecast  errors 
of  successive  forecasts . 


TIME  LAG  (HOURS) 

24-HR 

48-HR 

72-HR 

0 

1.000 

1 .000 

1 .000 

6 

.665 

.790 

.838 

12 

.432 

.587 

.675 

18 

.291 

.432 

.476 

24 

.213 

.305 

.371 

30 

.173 

.212 

.127 

36 

.181 

.171 

.177 

3.4  Discriminant  Analysis 

The  UCLA  Biomedical  computer  program  BMDP7M  (Dixon,  1975) 
was  used  to  attempt  to  discriminate  between  forecasts  likely  to 
produce  large  errors  and  those  likely  to  produce  small  errors. 
The  cases  were  classed  in  three  groups  according  to  their  known 
24-hour  errors.  Group  1 consisted  of  cases  where  the  magnitude 
of  the  error  was  less  than  the  median  in  both  W-E  and  S-N 
components.  Group  2 had  one  component  above  the  median  and  the 
other  below  the  median  while  Group  3 had  both  components  above 
the  median.  The  percentage  of  cases  falling  into  Groups  1,  2 
and  3 was  30t,  46%  and  24%,  respectively. 


The  di scrimi nators  made  available  for  the  analysis  were 
those  variables  previously  discussed  which  were  shown  to  be 
related  to  error  magnitude  and  would  be  known  to  the  fore- 
caster as  he  made  his  forecast.  These  were:  (1)  latitude; 

(2)  longitude;  (3)  maximum  wind;  (4)  number  of  storms  in 
progress;  (5  and  6)  the  two  components  of  motion  over  the 
previous  12  hours;  (7)  month;  and  (8)  distance  to  the  nearest 
"other"  storm,  if  one  was  present  otherwise  a large  default 
number  was  assigned.  Of  the  above  parameters,  the  first  six 
were  selected  by  the  analysis  program  as  significantly 
contributing  to  discrimination.^ 

Typically,  Group  1 was  characterized  by  well  developed 
tropical  cyclones  (typhoon  intensity)  in  the  western  part  of 
the  region  at  low  latitudes  and  moving  generally  west.  A 
Group  3 forecast  was  typical  during  or  after  recurvature  and 
included  the  total  spectrum  of  tropical  cyc.lone  intensities. 
Group  2 included  many  low  latitude  weak  depressions  or  tropical 
cyclones  of  all  intensities  at  or  near  typical  recurvature 
latitudes  but  not  as  yet  exhibiting  recurvature.  Also  included 
in  Group  2 were  many  otherwise  Group  1 cases  in  multiple  storm 
situations . 

The  forecast  error  distributions  are  illustrated  in 
Fig.  7 by  means  of  40%  probability  ellipses  bases  on  the 
assumption  of  a bivariate  normal  probability  distribution. 
Ellipses  are  given  for  each  group  for  24-,  48-  and  72-hour 
forecasts  although  the  discriminant  analysis  was  based  on 
24-hour  errors  only.  The  most  prominent  difference  in  the 
three  sets  of  ellipses  is  their  size.  The  area  within  each 


^The  resulting  classification  functions  are  shown  in 
Appendix  A . 

Q 

For  comparison  purposes , 70%  and  90%  probability  ellipses 
relate  to  distributions  2.55  and  2.15  timesy  respectively,  the 
axis  values  depicted  by  Fig.  7. 
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Figure  7.  The  40%  probability  ellipses  for  each  forecast 
interval  for  (a)  Group  1;  (b)  Group  2;  and  (c)  Group  3 
forecasts.  The  origin  is  the  forecast  position  with 
each  ellipse  center  the  average  verifying  position 
relative  to  forecast  position. 


of  the  Group  1 ellipses  is  roughly  half  that  of  the  corres- 
ponding Group  3 ellipses.  The  orientation  of  the  major  axis 
is  similar  for  all  three  groups. 

The  bias  or  offset  of  the  ellipse  center  from  the  diagram 
origin  is  striking  for  Group  2.  The  bias  to  the  southwest 
means  forecasts  were,  in  the  mean,  too  far  northeast.  This 

7 

could  be  the  result  of  over  anticipation  of  recurvature. 

Although  the  bias  is  much  less  for  Group  3,  it  is  in  the 
opposite  direction  perhaps  due  to  underestimation  of  the 
acceleration  of  recurved  tropical  cyclones.  The  bias  is  small 
but  slightly  to  the  west  for  Group  1.  Since  these  are  basic- 
ally east  to  west  moving  tropical  cyclones,  this  would  indicate 
speed  is  in  the  mean  slightly  underforecast. 

The  1976  cases  were  tested  for  fit  into  various  probability 
ellipses  constructed  on  the  dependent  data.  Figure  8 shows  the 
results  of  this  test.  In  each  of  the  small  graphs  the  45°  line 
represents  the  expected  result  while  the  dots  connected  by  line 
segments  represent  the  observed  fit.  The  maximum  deviation  is 
indicated  by  a vertical  arrow.  Using  the  Kol omogorov-Smi rnov 
test  (Massey,  1951)  based  on  the  effective  number  of  independent 
cases,  none  of  these  differences  are  significant  at  the  5^ 

Q 

level.  The  ellipses  were  generally  conservative  because  1976 
forecasts  were  better  than  the  10-year  average  perhaps 
reflecting  a general  improvement  of  forecasts  over  the  eleven 
years  involved. 


7 

Aa  an  example  for  a avngle  year,  the  1967  tropical 
cyclone  season  had  many  east-west  moving  tropical  cyclones 
with  below  average  number  of  recurving  tropical  cyclones , 

The  ?2-hour  forecast  error  bias  was  172  km  (93  n mi)  to  the 
southwest  for  Group  2 cases. 

Q 

Because  of  high  autocorrelation,  the  sample  size  was 
reduced  by  1/4  to  account  for  sample  dependence  (after  Brooks 
and  Carruthers,  1952). 
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Figure  8.  The  1976  verifying  positions  that  fell  into  25, 
50,  75,  90  and  95%  probability  ellipses  by  Group  and 
Forecast  (24,  48,  and  72  hours).  Maximum  deviation  is 
indicated  by  vertical  arrow. 


For  contrast  the  Group  3 and  Group  1 forecasts  were 
interchanged  and  the  ellipses  were  again  tested.  Figure  9 
shows  the  results  of  this  test.  The  Group  1 forecast  errors 
are  clearly  not  represented  well  by  the  large  Group  3 ellipses 
and  conversely  the  Group  3 forecasts  are  poorly  represented 
by  the  Group  1 ellipses. 

3.5  Annual  Variability 

Perhaps  the  most  interesting  facet  of  the  examination 
of  tropical  cyclone  forecast  errors  is  the  extent  of  annual 
variability  and  possible  reasons  for  the  variations.  Figure 
10(a)  shows  the  annual  mean  errors  for  the  11  years  1966-1976. 
The  picture  has  been  described  as  a steady  improvement  to  a 
minimum  in  errors  in  1970,  then  an  unsteady  but  generally 
worsening  trend  in  the  years  since.  The  least  squares  trend 
lines  indicate  minor  improvement  over  the  total  period,  however, 
the  slopes  are  too  shallow  to  be  conclusive. 

A number  of  factors  have  been  examined  to  isolate  reasons 
for  the  large  year  to  year  variability.  The  performance  of 
objective  forecast  techniques  tends  to  parallel  that  of  the 
official  forecasts.  Figure  10(b)  shows  the  annual  variation 
of  simple  linear  extrapolation,  or  persistence  as  a forecast 
technique.  The  persistence  error  depicts  a measure  of  how  well 
behaved  or  linear  the  tracks  were.  This  extrapolation  is  based 
on  "best  track"  or  post  analysis  positions.  Of  course  the 
forecaster  faces  uncertainty  in  the  location  and  recent  history 
of  the  tropical  cyclone,  nevertheless,  the  similarity  in  the 
twv  curves  illustrates  the  vulnerability  of  the  forecasting 
system  to  major  track  or  speed  changes. 

Another  measure  of  variability  is  the  type  of  forecasts 
which  make  up  a year.  That  is,  the  percentage  of  the  year 
made  up  of  the  type  expected  to  give  large  errors  versus  those 
expected  to  give,  small  errors.  In  Fig.  10(c),  the  annual  mean 
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Figure  9.  The  1976  verifying  positions  that  fell  into  25, 
50,  75,  90  and  95%  probability  ellipses  for  Group  1 into 
Group  3 ellipses  and  Group  3 into  Group  1 ellipses  for 
24,  48,  and  72  hours.  Maximum  deviation  is  indicated  by 
vertical  arrow. 
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Figure  10 . The  annual  variation  of  (a)  mean  tropical 
cyclone  official  forecast  error;  (b)  mean  persistence 
error;  (c)  mean  Group  number;  (d)  percentage  of  warnings 
based  on  reconnaissance  and  satellite;  and  (e)  number  of 
JTWC  duty  officers. 


group  numbers  are  shown.  This  comes  from  the  discriminant 
analysis  previously  described  where  Group  1 represents  a 
relatively  easy  forecast  and  Group  3,  a difficult  forecast. 

For  example,  1970  was  dominated  by  Group  1 forecasts  and  the 
mean  group  ..  )er  is  well  below  2.0.  The  1975  season  was 
dominated  by  the  difficult  Group  3 forecasts  and  the  mean  group 
number  is  above  2.0.  The  general  characteristics  of  the  curves 
of  Fig.  10(c)  follow  those  of  Fig.  10(a),  particularly  from 
approximately  1968  or  1969  to  1976. 

Figure  10(d)  shows  the  percentage  of  warnings  issued  by 
the  JTWC  based  on  aircraft  reconnaissance  and  satellite  data. 
There  is  qualitative  correspondence  between  large  percentages 
of  warnings  based  on  reconnaissance  in  the  middle  years  and 
generally  lower  forecast  errors.  Forecast  errors  are  larger 
with  the  lower  percentage  of  warnings  based  on  reconnaissance 
data  in  the  early  years  and  the  later  period.  In  the  years 
after  1971  satellite  data  has  to  some  extent  replaced  recon- 
naissance, but  as  indicated  previously  initial  position  error 
is  highly  related  to  forecast  error  and  the  initial  position 
error  for  reconnaissance  is  less  than  for  satellite  data.  For 
1976  the  initial  position  error  was  32.8  km  (17.7  n mi)  for 

g 

aircraft  reconnaissance  versus  56.5  km  (30.5  n mi)  for  DMSP. 

It  seems  apparent  that  the  reduction  of  aircraft  reconnaissance 
has  had  an  impact  on  the  forecast  errors.  In  addition,  in  the 
years  since  1971,  aircraft  reconnaissance  in  the  western  North 
Pacific  has  not  had  the  use  of  the  powerful  APS-20  radar  which 
provided  excellent  tropical  cyclone  "stand-off"  fix  capability. 


It  should  also  be  pointed  out  that  the  DMSP  program  has 
been  particularly  beneficial  or  cost  saving  in  the  reduction 
of  investigative  aircraft  missions  which  are  flown  when  the 
development  or  formation  of  a tropical  cyclone  is  imminent. 
Investigative  missions  have  been  cut  over  the  last  four  years 
from  about  four  to  one  per  tropical  cyclone ^ which  translates 
into  an  annual  savings  of  nearly  $1  million  (Pilipowskyj , 1977a). 
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Another  factor  which  has  changed  at  the  JTWC  in  recent 
years  is  numbers  of  duty  officers  or  forecasters.  The  number 
of  forecasters  has  historically  been  six,  the  Director, 
Operations  Officer  and  four  duty  officers.  As  can  be  seen  by 
Fig.  10(e),  the  number  of  duty  officers  present  during  the  year 
has  been  increased  in  recent  years.  It  is  certainly  not  clear 
that  this  increase  has  led  to  an  improvement  in  forecasting. 

But  the  question  is:  Has  there  been  an  improvement  in  fore- 
casting in  recent  years?  This  will  be  addressed  next. 

One  of  the  major  objectives  of  this  study  was  to  examine 
the  longer  term  change  in  the  forecasting  system.  This  is 
useful  not  only  as  a basis  for  evaluating  the  effectiveness  of 
the  system,  but  serves  a more  important  purpose.  Over  the 
next  few  years  the  tropical  cyclone  warning  system  in  the 
western  North  Pacific  will  rely  increasingly  on  dynamic  models 
as  a major  input.  It  is  then  important  that  a baseline  be 
established  for  relatively  short  term  evaluation  of  the  system, 
hopefully  to  measure  improvements,  but  also  as  a guard  against 
system  degradation. 

To  establish  such  a baseline  the  year  to  year  performance 
must  be  normalized  or  adjusted.  Annual  mean  error  is  a poor 
measure  of  performance  because  clearly  some  years  are  heavily 
loaded  with  difficult  forecasts  and  others  weighted  toward 
easier  forecasts.  As  a normalizing  tool,  a measure  of  diffi- 
culty is  necessary.  The  distribution  of  forecasts  into  the 
three  groups  presented  earlier  is  one  possible  measure.  The 
1970  season  was  heavily  dominated  by  the  relatively  easy  Group 
1 forecasts,  and  1975  by  the  difficult  Group  3 forecasts.  The 
mean  group  number  explains  about  30*  of  the  year-to-year 
variance  in  mean  error.  Another  important  component  of  error 
is  how  persistent  the  tracks  were.  The  annual  persistence 
error  explains  from  30  to  50*  of  the  annual  variance.  Clearly 
these  two  measures  are  not  independent.  Their  correlations 
decrease  in  time  from  about  0.50  at  24  hours  to  0.25  at  72 
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hours.  Another  Important  component  of  error  is  the  error  in 
the  initial  position.  This  error  is  assumed  to  he  independent 
of  the  two  previous  measures.  Initial  position  error  has  been 
documented  on  a case  by  case  basis  since  1971  and  annual  means 
are  available  since  1970.  There  are  other  factors  such  as  the 
introduction  of  objective  techniques,  increased  knowledge 
concerning  the  behavior  and  characteristics  of  tropical 
cyclones,  improvements  in  the  mid-latitude  prognostic  models, 
data  (synoptic  and  satellite)  availability,  and  changes  in 
personnel.  These  are  less  tangible  and  will  for  the  moment  be 
i gnored . 

A system  of  handicapping  the  annual  errors  has  been 
devised  to  adjust  for  annual  forecast  difficulty  and  the 
variability  in  initial  position  error  or  more  specifically 
reconnaissance  support.  The  first  component  of  the  handicap- 
ping system  is  based  on  a two  predictor  least  squares 
regression  equation.  The  predictand  is  annual  error  (AE  in 
km)  and  the  predictors  are  the  annual  mean  group  number  (G) 
and  the  annual  mean  persistence  error  (P  in  km)  as  follows; 


AE24  * 

-76.4  + 0.779  P24  + 

79.1  T24, 

(1) 

II 

00 

UJ 

< 

-86.5  + 0.748  P^g  + 

112.8  G^g,  and 

(2) 

AE72  » 

-227.0  + 0.738  P^g  ^ 

1-  207.1  G^2 

(3) 

These  equations  explain  43,  62  and  79%  of  the  variance  in  the 
annual  mean  errors  at  24,  48  and  72  hours,  respectively. 

The  second  component  of  the  handicap  is  based  on  the 
percent  of  forecasts  issued  where  the  initial  position  was 
based  on  aircraft  reconnaissance.  This  percentage  is  known 
for  the  11-year  period.  This  percentage  was  converted  to  an 
equivalent  annual  mean  initial  position  error  by  assigning 
33.3  km  (18  n mi ) as  the  initial  position  error  for  reconnais- 
sance based  tropical  cyclone  forecasts  (Harrison,  1975  and 
Pilipowskyj,  1977b)  and  using  the  known  7 years  of  initial 
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position  errors  to  deduce  the  initial  position  error  value 
based  on  "other"  types  of  fixes.  These  "other"  types  include 
land  and  ship  radar,  satellite  data,  extrapolation  and  synoptic 
data.  This  average  was  found  to  be  64.8  km  (35  n mi ) . The 
relationship  between  forecast  error  (E  in  km)  and  initial 
position  error  (Eq  in  km)  as  was  shown  graphically  in  Fig.  6(b) 
is  as  follows: 

E24  = 154.3  + 1.6  Eq  = 154.3  + 1.6  (33. 4R  + 64.9  (1.0-R)),  (4) 

E^g  = 363.2  + 2.4  Eg  = 363.2  + 2.4  (33. 4R  + 64 . 9 ( 1 . 0-R ) ) , (5) 

and 

E72  = 617.0  + 2.0  Eg  = 617.0  + 2.0  (33. 3R  + 64 .8 ( 1 . 0-R) ) , (6) 

where  R is  the  fraction  of  warnings  during  the  year  that  were 
based  on  aircraft  reconnaissance. 

The  two  handicaps  were  adjusted  by  subtracting  out  the 
minimum  annual  value.  The  1973  season  for  instance,  the  least 
difficult  year  based  on  Eqs.  (1)  through  (3),  was  assigned  a 
zero  handicap  for  difficulty.  In  a year  when  100%  of  the 
forecasts  were  based  on  aircraft  reconnaissance,  a zero  handi- 
cap for  reconnaissance  would  have  been  assigned.  Finally  the 
annual  mean  errors  were  reduced  by  the  sum  of  the  two  handicap 
components.  Figure  11  is  the  "unexplained  error."  The  trend 
is  more  consistent  than  the  error  trend  of  Fig.  10(a)^^  and 
reflects  in  part  the  "pay-off"  or  result  from  research  efforts, 
i.e.,  improved  forecasting  aids  and  numerical  models,  and  a 
better  understanding  of  the  tropical  cyclone  and  its  surround- 
ing environment.  Also  in  the  trend  line  is  the  result  of  more 
realistic  manning  of  the  JTWC  in  terms  of  the  number  of  fore- 
casters as  well  as  any  change  in  the  quality  or  skill  level  or 


^^The  adjusted  evvor-year  aorretation  is  approximately 
0,6  0.8  compared  to  the  actual  error-year  correlation  of  0.3 
for  Fig , 10(a) . 
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even  motivation  of  the  personnel.  Additionally,  a rather 
steady  improvement  in  satellite  support  no  doubt  deserves  some 
of  the  credit.  The  loss  of  ship  reports  with  the  post-Vietnam 
drawdown  may  have  contributed  negatively  to  this  trend  also. 

There  has  long  been  concern  over  the  rapid  turnover  of 
personnel  at  the  JTWC.  The  tour  length  is  two  years.  A new 
Director,  who  is  a dominating  force  in  the  forecast  system, 
was  on  hand  for  the  1967  season  and  for  each  odd  year  there- 
after. The  apparent  sawtooth  pattern  in  Fig.  11  supports  this 
concern.  For  the  five  two-year  tours,  the  24-hour  errors  were, 
on  the  average,  10%  greater  in  the  odd  or  first  year  as  com- 
pared to  the  even  years.  The  48-  and  72-hour  forecasts  showed 
similar  but  less  distinct  numerical  changes. 

In  order  to  establish  a measure  of  the  forecasters 
involvement  in  forecast  errors,  a questionnai re  was  sent  to  the 
past  five  Directors  of  the  Joint  Typhoon  Warning  Center  and 
the  results  are  presented  in  Appendix  B. 


4.  SUMMARY  AND  RECOMMENDATIONS 


In  light  of  the  objectives  outlined  earlier,  for  the  most 
part,  the  goals  of  the  study  have  been  attained.  It  has  been 
demonstrated  that  a small  number  of  readily  available  param- 
eters can,  with  reasonable  effectiveness,  classify  a tropical 
cyclone  forecast  as  likely  resulting  in  either  markedly  above 
or  below  average  errors.  Group  1 forecasts  have  a high  prob- 
ability of  below  average  errors  with  a low  probability  of 
above  average  errors.  Group  2 forecasts  have  approximately 
equal  probabilities  of  being  above  or  below  average.  Group  3 
forecasts  have  a low  probability  of  below  average  errors  with 
a higher  probability  of  above  average  errors.  It  was  found 
that  the  error  probability  ellipses  for  all  three  groups  are 
oriented  approximately  the  same  (major  axis  - northeast  to 
southwest),  but  the  area  of  the  Group  1 ellipses  was  roughly 
half  of  the  area  of  the  Group  3 ellipses.  In  addition  the 
year-to-year  variation  in  forecast  errors  over  the  past  11 
years  in  the  western  North  Pacific  appears  to  show  continued 
improvement  once  the  errors  are  adjusted  for  the  annual  biases 
that  are  present  which  contribute  either  negatively  or  posi- 
tively to  the  forecast  errors. 

The  examination  of  additional  parameters  could  improve 
the  identification  of  potentially  poor  forecasts.  Initial 
position  error,  which  was  shown  to  be  directly  related  to 
forecast  errors,  was  not  introduced  as  a discriminator  because 
it  is  not  generally  known  to  the  forecaster  at  the  time  of  the 
forecast.  In  addition,  the  synoptic  patterns  associated  with 
the  tropical  cyclone  has  not  been  considered.  Parameters  which 
are  associated  with  such  features  as  the  Tropical  Upper 
Tropospheric  Trough  (TUTT),  subtropical  ridge  and  transient 
troughs  in  the  westerlies  might  prove  to  be  important  discrim- 
inators especially  since  they  many  times  relate  to  the  basic 
problem  of  tropical  cyclone  recurvature. 
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Two  primary  sources  of  error  are  implicit  from  the 
statistical  analysis:  (1)  Those  related  to  the  recurvature 
phenomena  --  whether  to  forecast  recurvature  or  not  and  in  the 
speed  of  motion  after  recurvature;  and  (2)  those  errors  asso- 
ciated with  initial  positioning.  Research  aimed  at  these 
sources  offers  considerable  potential  for  improvement  in 
tropical  cyclone  forecasting.  The  greatest  hope  for  the 
solution  of  the  recurvature  problem  may  lie  with  dynamic 
models,  but  synoptic  and  statistical  studies  should  not  be 
ignored . 

The  positioning  problem  suggests  improvement  in  the 
reconnaissance  system.  While  more  and  better  equipped  aircraft 
is  a possible,  and  perhaps  expedient  solution,  developments  in 
the  satellite  area  may  be  the  ultimate  solution.  This  may 
involve  finding  a different  and  more  conservative  way  to  define 
the  location  of  a tropical  cyclone  other  than  the  so-called  eye 
or  estimated  center. 

A further  application  of  the  statistical  base  presently 
established  would  be  to  derive  "threat"  or  "strike"  probabili- 
ties for  specific  locations  in  the  western  North  Pacific.  For 
example.  Fig.  12  gives  the  probability  integrated  over  an  area 
139  km  (75  n mi ) to  the  left  and  93  km  (50  n mi ) to  the  right 
of  Kadena  Air  Base,  Okinawa  relative  to  the  forecast  track  of 
Typhoon  Fran.  The  integration  is  also  performed  over  time 
from  that  initial  time  of  1800  GMT,  7 September  1976  to  a time 
12,  24,  48  and  72  hours  later. 

In  addition,  another  application  would  be  to  derive  fore- 
cast probability  ellipses  (such  as  50,  75  and  95%)  for  specific 
forecast  positions  out  to  72  hours.  This  could  even  be  carried 
a step  further  by  adding  information  such  as  the  radius  to  the 
15.4  ms"^  (30-kt)  wind  or  3.7  m (12-ft)  sea  to  attain  a 
probability  for  these  parameters  to  aid  tropical  cyclone 
avoidance  and  decision  making. 
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Figure  12.  The  "strike"  probability  of  Typhoon 
Fran  (1800  GMT,  6 September  1976)  passing 
through  the  shaded  area  relative  to  Kadena  Air 
Base,  Okinawa.  The  probcibilities  are  based  on 
the  forecast  track  and  the  error  distributions 
for  this  type  of  tropical  cyclone.  Typhoon 
Fran  did  in  fact  miss  the  shaded  area  as  the 
storm  passed  to  the  east  of  Okinawa. 
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APPENDIX  A 


The  discriminate  analysis  resulted  in  three  functions 
which  are  linear  combinations  of  six  variables  as  follows: 


COEFFICIENTS 

VARIABLE 

FUNCTION  1 

FUNCTION  2 

FUNCTION  3 

Latitude  (deg) 

-0.06439 

0.08352 

0.04238 

Longitude  (deg) 

Maximum  Wi nd  (ms"^ ) 

S-N  Movt  (ms"^ ) 

W-E  Movt  (ms"^ ) 

-0.03231 

0.00274 

-0.03928 

0.04371 

0.02694 

-0.00239 

-0.15719 

0.20958 

-0.06587 

-0.04932 

-0.18310 

0.01311 

Nr.  Storms  in  Progress 

-0.26096 

0.01186 

1 .05585 

Constant 

4.66511 

-3.81310 

3.14344 

The  values  of  the  functions  at  the  group  mean  for  each  variable 
are  as  follows: 


FUNCTION  1 FUNCTION  2 FUNCTION  3 

Group  1 0.36059  0.05017  -0.01138 
Group  2 -0.04093  -0.07138  0.01688 
Group  3 -0.36186  0.07454  -0.01822 


In  order  to  determine  which  group  a forecast  would  fall 
into,  each  of  the  functions  f^  for  i = 1,  2,  3 are  evaluated 
using  the  known  actual  values  for  the  six  variables.  A 
squared  distance  is  then  determined  from  the  group  mean  value 
(u^  1^)  for  the  3 functions  (i)  and  the  three  groups  (k  = 1,2,3) 
with  the  following  equation. 


The  forecast  is  assigned  to  the  group  (k)  corresponding  to 
2 

the  minimum  0|^  value. 


APPENDIX  B 


One  parameter  that  is  difficult  to  quantify  is  the  "man" 
or  "woman"  involved  in  the  day  to  day  forecasting  of  tropical 
cyclones.  In  order  to  establish  a measure  of  his  or  her 
involvement  in  the  forecast  errors,  a questionnaire  was 
forwarded  to  the  past  five  Directors  of  the  Joint  Typhoon 
Warning  Center  and  the  responses  are  summarized  below: 

1 . How  long  does  it  take  to  become  a "very  good" 
tropical  cyclone  forecaster  (typhoon  du ty  officer)? 

The  range  of  answers  was  extremely  large  --  from 
two  years  to  eight  years.  Although  the  term  "very  good"  can 
be  interpreted  a number  of  ways,  most  felt  that  this  level 
could  be  closely  achieved  in  3-4  years  if  the  individual 
really  worked  hard  on  developing  his  or  her  proficiency. 

2 . Does  ability  continue  to  improve  or  level  off? 

The  general  response  indicated  that  if  the  indivi- 
duals motivation  and  effort  remained  high,  improvement  would 
be  dramatic  the  first  2-3  years  and  then  taper  off.  Since 
ability  was  related  to  experience,  the  ability  to  predict  the 
unusual  event  would  continue  to  increase. 

3 . What  are  the  characteristics  and  backgrounds  of  the 
"better"  tropical  cyclone  forecasters? 

The  characteristics  would  include:  (a)  a high  level 
of  motivation,  perseverance,  perfectionism  and  imperturb- 
ability; (b)  the  ability  to  integrate  data  from  a variety  of 
sources  into  a forecast;  (c)  an  avid  reader  of  technical 
literature  with  the  willingness  to  test  and  evaluate  new 
techniques;  and  (d)  the  ability  to  understand  the  limitations 
of  objective  techniques  and  numerical  or  dynamic  model  output. 
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The  background  should  include  good  training  and 
operational  experience  in  tropical  meteorology  or  3-5  years 
operational  experience  in  extratropical  meteorology.  Formal 
education,  prior  research  experience  and  a technical  background 
in  aircraft  reconnaissance  or  satellite  data  applications  would 
be  relevant  but  less  important. 

4 . Would  permanent  tropical  cyclone  forecasters  do  a 
better  job  than  the  present  two  year  military  tour? 

Four  out  of  5 answers  indicated  "yes,"  with  one 
suggestion  for  having  a 3 season  military  tour  which  would  be 
an  improvement  from  the  present  system  and  with  some  exceptional 
duty  forecasters  extended  to  a fourth  season.  Another  suggested 
having  a permanent  chief  forecaster  (civilian)  to  maintain 
continuity,  to  develop  extensive  expertise  and  to  train  newly 
assigned  forecasters. 


